Frame rate up-conversion

ABSTRACT

Content comprising a sequence of original frames at a first frame rate is up-converted to a second, higher, frame rate. Intermediate frames are calculated for temporal positions spaced between temporal positions of the original frames. An output sequence at the second frame rate comprises a repeating pattern of frames and, within each repetition of the pattern, there is: at least one original frame; at least one intermediate frame which is repeated; and at least one frame which is not repeated. The method is suitable for applications where an up-conversion factor is required in the frame rate, such as an up-conversion factor of five to up-convert from the 24 Hz frame rate used for film-based content to the 120 Hz frame rate used by displays. The output sequence can use all of the original frames to reduce detail flicker.

PRIORITY INFORMATION

This patent application claims priority from PCT patent applicationPCT/IB2009/053110 filed Jul. 17, 2009, which claims priority to EPpatent application 08104851.4 filed Jul. 23, 2008, both of which arehereby incorporated by reference.

BACKGROUND OF THE INVENTION

This invention relates to converting the frame rate of content to ahigher frame rate.

A range of different frame rates are used in the distribution anddisplay of content. Film-based content (e.g. made-for-cinema films)typically has a frame rate of 24 Hz. Video content, such asmade-for-television content, typically has a frame rate of 50 Hz or 60Hz. A frame rate of 50 Hz or 60 Hz is typically used for thedistribution of content, such as broadcast systems and recorded media.

Content with a frame rate of 24 Hz can be up-converted to the 60 Hzframe rate used for distribution by a process known as 3:2 pulldown(also known as 2:3 pulldown). This is shown in FIG. 1. An originalsequence of film frames A-E at a rate of 24 Hz is shown on the upperline. The lower line shows a sequence of frames output at 60 Hz. FramesA, C and E are each shown three times, while frames B and D are eachshown twice. Each frame in the pulldown sequence is a copy of theoriginal frame. The 3:2 pulldown sequence gives a rate of 2.5 times theoriginal frame rate, i.e. 2.5×24 Hz=60 Hz. However, the repetition offrames, and uneven rate at which frames are repeated, can be noticed bya viewer as moving objects do not follow their correct temporalposition. This deficiency is known as motion judder.

Many displays now use a higher frame rate of 100 Hz or 120 Hz in orderto reduce motion blur. The input content can be video content at a rateof 50/60 Hz or film content, which can either be received at a rate of24/25/30 Hz or at a rate of 50/60 Hz, with the original 24 Hz frame ratehaving been up-converted to 50 Hz/60 Hz. The problem of up-convertingfilm content which originated at 24 Hz, and which needs to be displayedat a higher rate of, for example, 120 Hz will be addressed here.

FIG. 2 shows one possible way of up-converting film-based content from24 Hz to 120 Hz. In FIG. 2 time is shown along the horizontal axis andthe position of a moving object is shown along the vertical axis.Straight line 20 indicates the true path of the moving object. At timet=0 the object is at the position labelled A, in frame A. At time t=1,the object is at the position labelled B, in frame B. The originalframes A, B of the film (i.e. the original frames which occurred at the24 Hz rate) are extracted and displayed. Motion estimation andmotion-compensation techniques are then applied to calculate fourintermediate frames I1-I4. For example, interpolation techniques can beused to calculate the intermediate frames by using the original frames Aand B. The position of the object, as calculated at the times 0.2, 0.4,0.6, 0.8, is also shown. As the objects lie on a straight line 20between the two original positions A, B the viewer sees a smooth movingobject. While the method shown in FIG. 2 can provide a smooth output atthe higher frame rate, it can require significant hardware resources andmay require an unacceptably long processing time (high video latency).This can make the technique unsuitable for certain applications,especially where a rendering device has limited resources such asmemory, memory bandwidth, processing budget and power (e.g. batterylife).

FIG. 3 shows another possible way of up-converting the frame rate offilm content to 120 Hz which requires fewer resources of a renderingdevice. Motion compensated processing up-converts the frame rate to 60Hz and then each generated frame at the 60 Hz rate is repeated, to givethe output frame rate of 120 Hz. In FIG. 3, original frame A isdisplayed at t=0 and again at t=0.2. A first intermediate frame I1,calculated with the temporal position of the object at t=0.4, is outputat t=0.4 and t=0.6. A second intermediate frame I2, calculated with thetemporal position of the object at time t=0.8, is output at t=0.8 andt=1.0. This allows up-conversion with less resources, and lower latency.A drawback is that the moving object is not perceived as sharply (motionblur). Another drawback is that one out of every two original framescannot be shown. In this example, frame B is lost because the object inframe B is at the wrong temporal position. Another drawback is that highfrequencies at high contrast are perceived to flicker. The sharporiginals and less sharp interpolated objects flicker at the low repeatfrequency (12 Hz).

The present invention seeks to overcome at least one of thesedisadvantages.

SUMMARY OF THE INVENTION

Accordingly, a first aspect of the present invention provides a methodof increasing the frame rate of content, the content comprising asequence of original frames at a first frame rate, the methodcomprising:

receiving the original frames;

calculating intermediate frames for temporal positions spaced betweenthe temporal positions of the original frames;

outputting a sequence of frames at a second frame rate, higher than thefirst frame rate, wherein the output sequence comprises a repeatingpattern of frames and, within each repetition of the pattern, there is:

-   -   at least one original frame;    -   at least one intermediate frame which is repeated; and,    -   at least one of the frames is not repeated.

Embodiments of the invention have an advantage of providing an output atthe second frame rate which offers a higher quality viewing experiencecompared to methods which repeat every frame, while also having anadvantage of requiring less resources (e.g. processing budget, memory,power) compared to methods which calculate a full set of intermediateframes.

Embodiments of the present invention are particularly advantageous forapplications where an odd up-conversion factor, such as an up-conversionfactor of five, is required in the frame rate (i.e. output frame rate isfive times higher than the input frame rate). This has a particularlyadvantageous application in up-converting film-based content from a 24Hz frame rate to the 120 Hz frame rate used by displays, although theinvention is not limited to up-converting film-based content.

In some embodiments, the step of outputting a sequence of frames outputsall of the original frames, which can reduce the amount of computationrequired to calculate intermediate frames, and can reduce the visibilityof detail flicker to a viewer.

The original frames can be received as an input stream at the first rate(e.g. a 24 Hz stream where every frame is an original frame).Alternatively, the original frames may be received as a stream at a ratewhich is higher than the first rate, such as a 60 Hz stream which hasbeen created by a 3:2 pulldown process.

Further aspects of the invention provides apparatus for performing themethod and software for performing the method. The software can beprovided as computer-executable code which is tangibly embodied on anelectronic memory device, hard disk, optical disk or any othermachine-readable storage medium or it can be downloaded to a processingdevice via a network connection.

These and other objects, features and advantages of the presentinvention will become more apparent in light of the following detaileddescription of preferred embodiments thereof, as illustrated in theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention will now be described, by way ofexample only, with reference to the accompanying drawings, in which:

FIG. 1 shows a 3:2 pulldown technique for converting the frame rate offilm-based content;

FIG. 2 shows a technique of up-converting the frame rate of film contentby calculating an intermediate frame at each intermediate time a frameshould be displayed between original frames;

FIG. 3 shows a technique of up-converting the frame rate of film contentby calculating intermediate frames at intermediate times betweenoriginal frames and repeating frames;

FIGS. 4-6 show techniques for up-converting the frame rate of filmcontent in accordance with embodiments of the present invention.

FIG. 7 shows an embodiment of apparatus for performing the method ofFIGS. 4-6.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 4 shows a first method of up-converting film-based content to ahigher frame rate. As in FIG. 2, time is shown along the horizontal axisand the position of a moving object is shown along the vertical axis.For clarity, the horizontal scale is labelled in normalised units 0, 1,2 etc. For the case where up-conversion is from 24 Hz, the interval 0 .. . 1 represents 1/24 Hz (41.6 ms). Straight line 40 indicates the truepath of the moving object. At time t=0 the object is at the positionlabelled A, in frame A. At time t=1, the object is at the positionlabelled B, in frame B. The original frames A, B of the film (i.e. theoriginal frames which occurred at the 24 Hz rate) are extracted anddisplayed. Motion estimation and/or motion-compensation techniques arethen applied to calculate two intermediate frames I1, I2. For example,interpolation techniques can be used to calculate the intermediateframes by using the frames A and B. The position of the object iscalculated at times 0.3 and 0.7. The actual position of the object atthese times should lie along line 40. It can be seen that at time t=0.2,the frame I1 for position t=0.3 is displayed. At time t=0.4, the frameI1 for position t=0.3 is displayed again. At time t=0.6 the frame I2 forposition t=0.7 is displayed. At time t=0.8 the frame I2 for positiont=0.7 is displayed again. Only two interpolated frames I1, I2 need to becalculated. Each interpolated frame I1, I2 is displayed a first time,slightly before the time for which the position has been calculated, anda second time, slightly after the time for which the position has beencalculated. This embodiment outputs a pattern of the form:

-   -   A I1 I1 I2 I2 B . . .        where A and B are original frames and I1, I2 are intermediate        frames, and B is the first frame of the next repetition of the        pattern.

This pattern results in the interpolated frames displaying the object atpositions which lie evenly about the straight line 40 which representsthe true path of the moving object. Displaying each interpolated frametwice and the original frames once introduces some irregular restjudder. The method requires a reduced amount of computation compared tothe full motion compensated up-conversion method shown in FIG. 2, asonly two intermediate frames must be computed compared to four framesfor the method of FIG. 2. As all originals are shown, the detail flickerwill have a higher frequency of 24 Hz instead of 12 Hz and thus will beless visible or annoying to a viewer.

In this embodiment, and the following embodiments, particular values oftimes are quoted but it will be appreciated that the intermediate framescan be calculated at times which differ slightly from those describedand illustrated, without substantially affecting the effect of theinvention.

FIG. 5 shows a second method of up-converting film-based content to ahigher frame rate. As in previous figures, time is shown along thehorizontal axis and the position of a moving object is shown along thevertical axis. Straight line 50 indicates the true path of the movingobject. At time t=0 the object is at the position labelled A, in frameA. The original frames A-D of the film are extracted. In thisembodiment, not all of the original frames are displayed, but theoriginal frames are used to construct interpolated frames I1-I9.Original frame A is displayed at t=0 and original frame D is displayedat t=3. Frame D is the first frame of the next repetition of the patternof frames. Motion estimation and/or motion-compensation techniques areused to calculate intermediate frames I1-I9 for a set of temporalpositions between frames A and D. A frame is displayed at each 0.2increment along the time axis. The displayed sequence is a repeatingpattern of 1 frame followed by 1 repeated frame (i.e. 121212 . . . ).The original frame A is displayed at t=0 and again at t=0.2. Anintermediate frame I1 is displayed at t=0.4. The intermediate frame I1displayed at t=0.4 is a frame calculated for time t=0.3 as that is theposition where the eye expects the object. At t=0.6 an intermediateframe I2 is displayed which has been calculated for time t=0.6 and thisframe I2 is repeated at t=0.8. At t=1.0 an intermediate frame isdisplayed which has been calculated for the time t=0.9. The patterncontinues as shown in FIG. 5, with a frame being displayed at incrementsof 0.2 units, and repetition of alternate frames (1212 etc.). At t=3original frame D is displayed and the pattern repeats. Only one out ofthe three original frames are displayed, but intermediate frames aredisplayed at temporal positions very close to the original frames: aframe calculated for time t=0.9 is displayed at t=1 (the true positionof frame B); a frame calculated for time t=1.8 is displayed at t=2 (thetrue position of frame C). Interpolation techniques can be used tocalculate the intermediate frames I1-I9 by using original frames on eachside of the temporal position for which the intermediate frame isrequired. This embodiment outputs a pattern of the form:

-   -   A A I1 I2 I2 I3 I4 I4 I5 I6 I6 I7 I8 I8 I9 D D . . .        where A and D are original frames and I1-I9 are intermediate        frames and D is the first frame of the next repetition of the        pattern.

The pattern of FIG. 5 results in the intermediate frames displaying theobject at positions which lie evenly about the straight line 52, whichis displaced slightly from the line 50 which represents the true path ofthe moving object. The usage of a regular repeat pattern reduces theirregular rest judder. This method requires a reduced amount ofcomputation compared to the full motion compensated up-conversion methodshown in FIG. 2. The effect of the moved line is to introduce someregular rest judder. Compared to FIG. 4 this method swaps betweensingle/dual image more frequently and therefore the resulting“irregularity” is less annoying.

FIG. 6 shows a third method of up-converting film-based content to ahigher frame rate. Straight line 60 indicates the true path of themoving object. At time t=0 the object is at the position labelled A, inframe A. All of the original frames A-D of the film are extracted anddisplayed. Motion estimation and/or motion-compensation techniques areused to calculate intermediate frames I1-I7 for a set of temporalpositions between frames A and D. A frame is displayed at each 0.2increment along the time axis. The displayed sequence is a repeatingpattern of one frame followed by one repeated frame (i.e. 121212 . . .). The original frame A is displayed at t=0 and again at t=0.2. Anintermediate frame I1 is displayed at t=0.4. The intermediate frame I1displayed at t=0.4 is a frame calculated for the temporal positiont=0.3. At t=0.6 an intermediate frame I2 is displayed which has beencalculated for time t=0.6 which is repeated at t=0.8. At t=1.0 theoriginal frame B is displayed. At t=1.2 an intermediate frame I3 isdisplayed which has been calculated for time t=0.2. This frame isrepeated at t=1.4. At t=1.6 an intermediate frame I4 is displayed whichhas been calculated for the object being at temporal position t=0.5. Att=1.8 an intermediate frame I5 is displayed which has been calculatedfor the time t=0.8 and this is repeated at t=2. Original frame C isdisplayed at t=2.2. The pattern continues as shown in FIG. 5. Originalframe D is displayed at t=3 and then the pattern repeats as shown fort=0. As in FIG. 5, frames are displayed at increments of 0.2 units, andframes are calculated for temporal positions at increments of 0.3 units(0.3, 0.6; 0.2, 0.5, 0.8; 0.4, 0.7) and alternate frames are repeated.Interpolation techniques can be used to calculate the intermediateframes I1-I7 by using original frames on each side of the temporalposition for which the intermediate frame is required. Stated anotherway, this embodiment outputs a pattern of the form:

-   -   A A I1 I2 I2 B I3 I3 I4 I5 I5 C I6 I6 I7 D D . . .        where A-D are original frames and I1-I7 are intermediate frames        and D is the first frame of the next repetition of the pattern.

The pattern of FIG. 6 results in the intermediate frames displaying theobject at positions which lie evenly about the straight line 62, whichis displaced slightly from the line 60 which represents the true path ofthe moving object. The regular use of repeating frames reduces theamount of irregular rest judder. This method requires a reduced amountof computation compared to the full motion compensated up-conversionmethod shown in FIG. 2, and a reduced amount of computation compared toFIG. 5 as only seven intermediate frames (I1-I7) must be computedcompared to nine intermediate frames (I1-I9) for the method shown inFIG. 5. Displaying all of the original frames reduces the amount ofdetail flicker compared to FIG. 5.

FIG. 7 shows an embodiment of apparatus for performing the methoddescribed above. An input 71 receives frames in the form of, forexample, progressive scan frames of film content at a rate of 24 Hz orvideo frames at a rate of 60 Hz. The frames received at input 71 mayalready be in digital form, or they may be in analog foul and areconverted to digital form at the input 71. Received frames are stored inan input frame buffer 72. If the input to frame buffer 72 is at thehigher rate, unit 73 extracts original frames from the content. Theoriginal frames are forwarded 75 to an output stage 77 and to anintermediate frame calculation unit 74. Various techniques forextracting original frames are known to the skilled person. Unit 74calculates intermediate frames. Unit 74 can use interpolationtechniques, which make use of the original frames 75, or other motiondetection and motion compensation techniques which are known to theskilled person. Intermediate frames are calculated at temporal positionsas described in FIGS. 4-6. Intermediate frames are forwarded 76 tooutput stage 77. Output stage 77 constructs an output sequence offrames, making use of original frames 75 and the intermediate frames 76,in the manner shown in FIGS. 4-6. The output sequence is stored in anoutput frame buffer 78 and delivered to an output 79 at the output framerate, e.g. 120 Hz, or the output sequence can be output directly withoutbeing stored in the output buffer 78.

The apparatus shown in FIG. 7 can form part of a consumer electronicsdevice, such as a television, DVD player, Blu-ray player, media player,set-top box, display, video processing unit, or it can form part of aprocessing unit in any part of the distribution chain between a contentsource and a display.

It should be noted that the above-mentioned embodiments illustraterather than limit the invention, and that those skilled in the art willbe able to design many alternative embodiments without departing fromthe scope of the appended claims. In the claims, any reference signsplaced between parentheses shall not be construed as limiting the claim.The words “comprising” and “including” do not exclude the presence ofother elements or steps than those listed in the claim. Where thesystem/device/apparatus claims recite several means, several of thesemeans can be embodied by one and the same item of hardware.

The various illustrative logical blocks, modules, circuits, andalgorithm steps described above may be implemented as electronichardware, as software modules executed by a processor, or ascombinations of both. To clearly illustrate this interchangeability ofhardware and software, various illustrative components, blocks, modules,circuits, and steps have been described above generally in terms oftheir functionality. The various illustrative logical blocks, modules,and circuits described in connection with the embodiments disclosedherein may be implemented or performed with a general purpose processor,a digital signal processor (DSP), an application specific integratedcircuit (ASIC), a field programmable gate array (FPGA) or otherprogrammable logic device, discrete gate or transistor logic, discretehardware components, or any combination thereof designed to perform thedescribed functions. A general-purpose processor may be amicroprocessor, a conventional processor, a controller, amicrocontroller, or a state machine. A processor may also be implementedas a combination of computing devices, e.g., a combination of a DSP anda microprocessor, a plurality of microprocessors, one or moremicroprocessors in conjunction with a DSP core, or any other suchconfiguration.

A software module may reside in RAM memory, flash memory, ROM memory,EPROM memory, EEPROM memory, registers, hard disk, a removable disk, aCD-ROM, or any other form of storage medium known in the art. Anexemplary storage medium is coupled to the processor such that theprocessor can read information from, and write information to, thestorage medium.

In the description above, and with reference to the Figures, there isdescribed a method and apparatus for up-converting content comprising asequence of original frames at a first frame rate. Up-conversion is to asecond, higher, frame rate. Intermediate frames are calculated fortemporal positions spaced between temporal positions of the originalframes. An output sequence at the second frame rate comprises: at leastone original frame; at least one intermediate frame which is repeated;and at least one frame which is not repeated. The method is suitable forapplications where an odd up-conversion factor is required in the framerate, such as an up-conversion factor of five to up-convert from the 24Hz frame rate used for film-based content to the 120 Hz frame rate usedby displays. The output sequence can use all of the original frames toreduce detail flicker.

Although the present invention has been illustrated and described withrespect to several preferred embodiments thereof, various changes,omissions and additions to the form and detail thereof, may be madetherein, without departing from the spirit and scope of the invention.

What is claimed is:
 1. A method of increasing the frame rate of content,the content comprising a sequence of original frames, including a firstoriginal frame, a second original frame, a third original frame, and afourth original frame, at a first frame rate, the method comprising:receiving the original frames; calculating intermediate frames fortemporal positions spaced between the temporal positions of the originalframes; outputting a sequence of frames at a second frame rate, higherthan the first frame rate, wherein the output sequence comprises arepeating pattern of frames and, within each repetition of the pattern,there is: at least one original frame; at least a first intermediateframe calculated from at least two consecutive original frames, thefirst intermediate frame being repeated by at least a secondintermediate frame that is identical to the first intermediate frame;and at least a second intermediate frame that is not repeated.
 2. Amethod according to claim 1 wherein the second frame rate is an oddinteger multiple of the first frame rate.
 3. A method according to claim2 wherein the second frame rate is five times higher than the firstframe rate.
 4. A method according to claim 3 wherein the first framerate is 24 Hz and the second frame rate is 120 Hz.
 5. A method accordingto claim 1 wherein the outputting the sequence of frames comprisesoutputting all of the original frames.
 6. A method of increasing theframe rate of content, the content comprising a sequence of originalframes, including a first original frame, a second original frame, athird original frame, and a fourth original frame, at a first framerate, the method comprising: receiving the original frames; calculatingintermediate frames for temporal positions spaced between the temporalpositions of the original frames; outputting a sequence of frames at asecond frame rate, higher than the first frame rate, wherein the outputsequence comprises a repeating pattern of frames and, within eachrepetition of the pattern, there is: at least one original frame; and atleast one intermediate frame calculated from at least two consecutiveoriginal frames, each intermediate frame being repeated by at least asecond intermediate frame that is identical to the first intermediateframe.
 7. A method according to claim 1 wherein the step of outputtingthe sequence of frames comprises an alternating sequence of repeating aframe and outputting the next frame only once.
 8. A method according toclaim 7 wherein for each intermediate frame that is repeated, theintermediate frame is calculated for a temporal position at which atleast one occurrence of the intermediate frame will be output.
 9. Amethod according to claim 1 further comprising receiving a sequence offrames at a frame rate higher than the first frame rate and extractingthe original frames at the first frame rate.
 10. A method according toclaim 9 wherein the sequence of frames at a frame rate higher than thefirst frame rate comprise a sequence of frames which have beenup-converted using a 3:2 pull down process.
 11. A method according toclaim 6 wherein the second frame rate is five times higher than thefirst frame rate, the step of calculating intermediate frames calculatestwo intermediate frames between each adjacent pair of original frames,and the step of outputting the sequence of frames comprises repeatingeach intermediate frame and outputting each original frame only once.